Conventional systems tend to waste precious resources when communicating via a queue pair (QP) connection. A conventional QP includes a dedicated send queue (SQ) and a dedicated receive queue (RQ) as well as a dedicated or shared completion queue (CQ). Thus, two-way communications can be achieved between peer systems via respective QPs. For example, a message can be sent from an SQ of a first peer system to an RQ of a second peer system. Similarly, a message can be sent from an SQ of the second peer system to an RQ of the first peer system.
When the number of connections between the peer systems increases, the number of QPs increases. Some conventional peer systems allocate the same amount of resources of a preset size to support each QP including the respective dedicated SQ and the respective dedicated RQ. However, such over-provisioning may be wasteful, in particular, with respect to the dedicated RQs. For example, if a particular connection is inactive or relatively inactive, a substantial portion of the resources allocated to the RQ of the inactive or relatively inactive QP remains unused and is not re-allocated for other purposes.
RQ resources are provisioned in anticipation of a burst of messages at wire speed. The user of a transport layer (e.g., a transmission control protocol (TCP) layer) provides a wide-open receive Window to improve performance beyond, for example, 64 KB per connection, thereby forcing the upper layer protocol (ULP) layer to manage the burst situation. Many applications such as Internet Small Computer System Interface (iSCSI) applications do not have a good upper bound as to the number of unsolicited messages that can be sent by one peer and received by another. This may cause connections to be torn down. Managing RQ resources on a per QP basis implies that each QP has to have enough buffers allocated to its RQ to handle a potential burst. When a particular machine serves a high number of QPs, a lot of buffers are allocated to QPs that may utilize them at a very low rate.
Some conventional systems do not control which connections consume buffers and which connections are dropped. In layered protocol processing, when resources are scarce, data is buffered by a central entity such as a TCP layer until a ULP application procures additional buffering. In rototilled protocol processing, when resources are scarce, data or the connection is dropped requiring either resending of the data or re-establishing the connection. When the ULP application procures additional buffers, then the ULP application may again receive data. In either processing scheme, buffers are consumed on a first-come-first-serve basis. Thus, typically inactive or relatively inactive connections can consume a substantial portion of the available resources, thereby limiting the available resources for typically active connections. Such typically active connections, which tend to be the more important consumers, suffer from reduced traffic bandwidth, are torn down, or have higher costs associated with them for the same level of functionality, for example, as larger memories are provided.
Some conventional systems suffer from substantial delays in adjusting the amount of allocated resources for a particular connection. For example, in a credit-based system at least one round-trip delay between peer systems may be required before a particular resource allocation can be modified. Thus, if a connection suddenly becomes active thereby needing additional resources, then one peer system informs the other peer system which, in turn, acknowledges the change in resource allocation to the original peer system before communications using the adjusted resource allocation may commence. The round-trip delay can be substantial between distant peers or over a network with substantial latencies and may require additional attention from a software layer adapted to provide buffers. In many systems, the response time of such a software entity is large in comparison with network speeds and the response time required by high performance applications. In particular, the round-trip delay becomes compounded each time the resource allocation of the connection is adjusted. The round-trip delays also exacerbate synchronization issues between hardware and software. Moreover, as the number of connections increases, the inefficiencies in adjusting the resource allocations become accentuated.
Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of ordinary skill in the art through comparison of such systems with some aspects of the present invention as set forth in the remainder of the present application with reference to the drawings.